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ABSTRACT 

A group of nine persons met in Minneapolis at the 
invitation of the Minnesota Research Coordinating Unit to discuss 
some of the innovative empirical techniques for implementing the 
"systems" approach to vocational curriculum development, including 
some military pioneering efforts. The seven presentations discussed 
at the seminar included: (1) "A Systems Application to Curriculum 

Improvement" by David S. Bushncli, (2) "A Manpower Delivery System: 
Implications for Curriculum Development" by Robert G. Smith, Jr., (3) 
"Implications of Air Force Occupational Research for Curriculum 
Design" by Raymond E. Christal, and (4) "An Empirical Procedure for 
Identifying the Structure of the Technical Concepts Possessed by 
Selected Workers" by Jerome Moss, Jr. and others. Throughout the 
seminar, discussion was directed to (1) the preparation of students 
for entry or reentry into the labor market in relation to specifying 
the worker’s role and tasks, (2) the selection of tasks to be taught, 
analyzing the selected tasks and stating performance objectives, and 
(3) the specification of instructional sequence. A major outcome of 
the seminar was a framework for use in describing the nature of 
conceptual and operational problems facing curriculum builders. (OS) 
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Introduction 



On February 28 and March 1, 1970 a group of nine persons met in Minneapolis, 
Minnesota, at the invitation of the Minnesota Research Coordinating Unit 
for Vocational Education, to discuss the process and techniques of voca- 
tional curriculum development. The participants are listed on page vii. 

A tenth person who had been invited, Joe Silverman, Naval Personnel and 
Training Laboratory, was unable to attend but his paper is included in this 
Report . 

There were two primary and closely related reasons for conducting the semi- 
nar. First, new models of the curriculum development process based upon 
the "systems" approach, and innovative empirical techniques for carrying 
out the process are now being developed and tested. The vocational educa- 
tion community should be made aware of them. Second, a high percentage of 
these pioneering efforts are being carried out in and for the Armed Services. 
A closer relationship between occupational education researchers working 
in the military and those giving attention to non-military programs is 
long overdue — - undoubtedly to the detriment of both groups. The seminar 
was, therefore, perceived as a small beginning to what is hoped will be 
an expanding, continuing forum. 

The seminar was kept small so that its members could benefit from the maxi- 
mized opportunity for informal discussion. Papers were written and dis- 
tributed before the seminar to prepare participants for the two-day meeting. 

This Report is our means of communicating with that portion of the vocational 
education community who we feel are most directly concerned with the 
methodology of curriculum development. The papers are reproduced as pre- 
pared by their authors. The "Summary" has been prepared by the editors and 
is a severe./ abridged account of the actual discussion, which reflects the 
editors' biases and limited perceptions and perspectives. 

The staff of the Minnesota Research Coordinating Unit is very appreciative 
of the time and effort expended by the seminar members and contributors 
to this Report . We extend special thanks to Rupert Evans who chaired the 
seminar. 
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SUMMARY OF THE DISCUSSION 



Jerome Moss, Jr. and Brandon B. Smith 



The discussion spurred by each presentation was wide-ranging in its 
coverage of the curriculum development process. To present the summary 
in chronological fashion would be to attempt a linear description of a 
spiral-like treatment of the seminar topic. The editors have therefore 
elected to organize the summary around their perception of the curriculum 
development process as it began to evolve from the discussions. Special 
techniques, considerations and problems will be related to the particular 
step(s) in the process with which they were associated. In this manner, 
it is hoped that the summary will be of maximum value to readers. 

The organiza:ion selected, however, necessitated vising a considerable 
amount of editorial license. No attempt was made at the seminar to reach 
a consensus regarding the specific steps in the curriculum development 
process, nor even to discuss all of the steps. Thus, the framework em- 
ployed here is not necessarily entirely acceptable to each of the seminar 
participants. Certainly, it is not considered a description of the com- 
plete process by either the participants or the editors. 

Throughout the seminar, discussion of the curriculum development pro- 
cess was applied to programs designed to prepare students for immediate 
entry or reentry to the work force} application of the process to "pre- 
vocational" or "orientation to work" programs was not discussed. 



I. Specify the Role 



The first step in the curriculum development process needs to be the 
specification of the role for which training is to be provided. 

For the military, which frequently has foreknowledge of new weapons 
and other systems and which has considerable control over both occupational 
demand and supply, this step is fairly straightforward. A systems analysis 
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can be conducted to define the scope and function of each "job" that needs 
to be done within the system in order to accomplish the system's mission. 
Individuals, who most frequently do not make a career in the military, can 
then be trained for specific, well-defined jobs (work roles). 

For public school vocational educators, however, social values and 
practical considerations make role definition much more difficult. Voca- 
tional educators have little control over the labor supply, and almost 
none over occupational demand. Their programs are not the sole suppliers 
of the nation's trained labor. And vocational educators cannot be con- 
cerned only with satisfying manpower requirements, but must try to opti- 
mize the entire social system through maximizing the potential of every 
student. Consequently, the public school vocational educator must con- 
sider many factors, in addition to the needs of available specific "jobs," 
when defining the work roles for which training is to be provided. For 
example, in order to help assure individual job satisfaction, as well as 
satisfactory job performance, job engineering — the restructuring of 
work roles — may be required to adapt jobs to available human abilities 
and psychological needs (e.g. creating paramedical occupations). Consider 
also the effects of imposed, unrealistic labor market barriers, such as 
discriminatory practices, inefficient placement systems, and control by 
unions or management over entry points to certain career fields. To what 
extent should these barriers to the use of skills effect the scope and 
nature of the work roles for which training is to be provided? Further, 
concern for the career development of the individual also forces public 
school vocational educators to focus attention on the long-term usefulness 
of work skills. Studies of the predicted impact of technological change, 
the identification of occupational "clusters," and the specification of 
career ladders are all indicative of a desire to enhance the efficiency 
of training by defining a work role for training purpo? »s that is diffe r- 
ent from the entry level job currently available in the labor market. 

Finally, many vocational educators in the public schools would speci- 
fy the roles for which training is to be provided to include aspects of 
the non-work as well a*, the work roles. Concern for citizenship and 
culture-carrying activities and the self-fulfillment, of the individual 
would dictate this broader perspective of vocational education's responsi- 
bility. 



I I . Id entify the Tasks in the Specified Role 



After specifying the role, that is, putting boundaries around the 
total domain of concern for a given curriculum, it is then possible to 
identify the specific tasks that comprise the role. Tasks are groups of 
activities that generally occur together and which have a common purpose 
which is valuable in and of itself. Some examples of work tasks are: 
perform blood count, establish work priorities, plan work flow, make lo- 
cal purchase of supplies, audit cash payments, maintain files of plans 
and programs, receive and distribute computer outputs. 
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There are at least three kinds of problems attendant to the step of 
task identification: (a) What tasks within the role are educationally 

relevant? (b) What task sampling frame should be used? (c) What techniques 
of task identification are most efficient? These questions are discussed 
briefly below. 

Ideological questions , such as the purposes of occupational instruction 
in different institutions, at different educational levels, and for various 
occupational groups, might affect the kinds of role tasks that are relevant 
to the educator. For example, the technical tasks of the work role involving 
cognitive and psychomotor skill performance are typically considered rele- 
vant by all vocational educators. But what about the affective behaviors 
that are involved in occupational adjustment and human relations tasks on 
the job? Are these to be identified as work role tasks to be taught? [The 
military and non-military seminar participants tended to disagree on this 
question*] The same issue can be extended; whst kinds of tasks inherent 
in the citizenship and culture-carrying roles are to be included among 
the tasks to be taught ir. vocational curriculums? 

It was fully recognized by seminar participants that it might be ex- 
tremely difficult to identify all of the relevant tasks in some roles and/ 
or impractical to teach all the tasks after they had been identified. For 
instance, as work roles become more complex — as tasks involve greater 
degrees of Judgment rather than knowledge of procedure, as cognitive per- 
formance increases in relation to psychomotor performance, and as task 
generalization becomes important to performance *— the problem of relevant 
task identification is intensified. Similarly, as the number of relevant 
tasks grow, and as training time (and cost) is restricted, the feasibility 
of . teaching all identified tasks decreases. For these reasons, It is im- 
perative for the vocational curriculum developer to create and make ex- 
plicit a task sampling frame which classifies the total domain of relevant 
tasks from which specific tasks may be sampled for subsequent treatment 
and incorporation into the curriculum. There is no doubt that almost all 
non-mil itarv curriculum development efforts in the past have, in fact, 
sampled the tasks to be taught, but thus far none of them ha/e made the 
basis for their sampling explicit. They are, therefore, unable to defend 
their selection of particular tasks as representative of the total, rele- 
vant tack domain. 

It is quite likely that the appropriate sampling frames will need to 
be multi-dimension8l, There could be, for example, dimensions dealing 
with (a) work vs. non-work tasks, (b) entry Job vs. career development 
tasks, (c) specific vs. related Job tasks, (d) predominately cognitive vs. 
psychomotor vs. affective tasks, (e) tasks organized according to the hier- 
archical psychological processes required to learn or to perform them 
(per Gagne\ and so forth. 

Considerable time in the seminar was devoted to discussion of some 
techniques for identifying work role tasks. One successful task inventory 
technique (see the Christal paper in this R eport ) is to develop a task 
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check list which can then be submitted to workers in the appropriate role 
(job) for confirmation and for specification of the relative time spent on 
each task. The data yielded is apparently valid and reliable. By applying 
certain statistical clustering techniques to the data, job clusters based 
upon task similarity can be formulated. Experience with the technique 
has shown that it is fa,, superior to asking supervisors and former students 
about the value of prior training; asking about previous training tends 
to Identify missing training elements, but falls to Identify instructional 
"deadwood," that is, tasks taught but which are not being used on the Job. 
Presently, the task inventory technique lacks adequate means for measuring 
the proper performance conditions and level of each task, which is impor- 
tant to the subsequent specification of educational objectives. The tech- 
nique also becomes more difficult to apply as work roles increase in com- 
plexity and as fewer casks are exactly replicaole over time on the Job. 

One application of the task inventory technique that might facilitate 
its use with more complex jobs is to first devise a classification of tasks, 
e.g. decisions to be made, information needed for the decisions, etc., 
which then permits cross-checking across the task categories in an inter- 
view situation to help insure complete task identification (see the Ammer- 
man paper in this Report) . 

A second technique discussed by the seminar provided means for identi- 
fying the concepts and the relationships among concepts possessed by per- 
sons performing the work role at the desirable level of competence (sec 
the Moss and Pucel, et. al. paper in this Report ). The technique seems 
particularly suited to complex jobs in which instruction is less directly 
replicable on-the-job, but in which an interpretive and associative base 
for making judgements and applying concepts to a wide variety of tasks and 
situations is required. The technique, however, needs to be combined with 
some form of task identification method in order to be able to specify 'he 
instructional vehicles (tasks) needed to develop the designated conceptual 
structure. 

Although mest of the discussion dealt with techniques for identifying 
either cognitive or psychomotor tasks, the participants recognized the 
importance of being able to specify relevant affective tasks. It was pro- 
posed that these behaviors are simply another type of "task," and may, 
therefore, be treated after identification in the same way as other tasks. 
The immediate difficulty seems to be that we currently lack adequate 
methods, instruments and/or observation skills to reliably Identify affec- 
tive tasks and to specify the level of performance required for each. 



Ill, Select the Tasks to be Taught 



Since it is typically not practical to teach all of the relevant 
tasks comprising the specified role, criteria for selecting the tasks to 
be taught are required. The following are some of the potential criteria 
identified by the seminars (a) The practical limit on the number of tasks 
to be taught is a function of the total time available and the time it 
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takes to teach each selected task to a reasonable level of "functional 
utility"; it was agreed by the participants that teaching "about" tasks 
is not conducive to adequate task performance nor to task generalization, 
and is therefore an inefficient instructional approach, (b) Tasks must 
be selected from the task sampling frame so as to be representative of 
the total task domain, (c) Tasks should be selected that permit the 
greatest generalizability. These may be those tasks that most efficiently 
develop the desired conceptual structure (per Moss and Pucel, et al.); 
they are also the tasks that optimize the stimulus repertory, response 
repertory, response association, task repertory and self-directed tendency 
(per Altman paper in this Report ) ; they may be selected on the basis of 
a taxonomy (per the Silverman paper), (d) Tasks which have the greatest 
frequency of use in the role (or time spent on them) might be selected. 

(e) Tasks which have emergency value or grave consequences for poor per- 
formance should be selected, (f) Tasks with the lowest "perishability" 
rate might be selected, (g) Tasks that are more "economical" to learn 
in the. formal training program than on-the-job should be selected, (h) 
Tasks whose performance might reasonably be considered a prerequisite to 
the training program should not be selected. 

When information bearing on the above kinds of criteria becomes avail 
able, some means for weighting each piece of data for making a final judge 
ment about each task will be needed. In that event, a "policy capturing 
model" (per the Christal paper in this Report ) might prove to be very use- 
ful until an empirical weighting based upon actual student performance in 
the specified role can be developed. 



IV. Analyse Each of the Selected Tasks 



Very little time was spent in the seminar on this step since Gagne' 
has already suggested a suitable task analysis technique. It was noted, 
however, that if a conceptual map were available it could provide a basis 
for checking the comprehensiveness of the results of the subjective task 
analyses. 



V. State Performance Objectives 



Using Mager's technique for stating performance objectives and 
applying it to the tasks selected to be taught, terminal objectives may 
be prepared as the next step in the curriculum development process. The 
accomplishment of each task can be stated as an objective. [Of course, 
terminal objectives may also be stated in terms of acquiring an approp- 
riate cognitive structure. ) 

The results of analysing each selected task will yield the specific 
competencies necessary for stating intermediate or enabling performance 
objectives as they relate to each terminal objective. At the same time, 
results of the task analysis permit the specification of required compe- 
tencies for entrance to the training program. 
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The seminar was generally critical of many curriculum development 
efforts which appeared to begin with the preparation of written perfor- 
mance objectives; while the achievement of those objectives may be 
measured, they may have no real validity or usefulness in terms of actual 
role performance. The seminar was also quick to note that the model de- 
scribed herein has as yet incorporated no reliable technique for deter- 
mining the desired level of task performance at exit from the training 
program. The statement of performance objectives may therefore lack 
realistic performance standards. 



VI • Specify the Instructional Sequence 



Sequencing instruction as to achieve the required level of performance 
of all selected tasks in the most efficient manner is a difficult problem 
that has not received the attention it warrants. 

At the micro level of sequencing activities, principles for generali- 
zation, such as those proposed by Altman in this Report , serve to suggest 
efficient reinforcement schedules. The task analysis process itself will 
yield sequences of sub-tasks which indicate instructional order. But the 
curriculum process must provide means to order and organize these other- 
wise disconnected individual task sequences. The seminar briefly discussed 
several possible techniques. 

First, if the work role is relatively simple and "proceduralized," 
the tasks selected to be taught can logically be presented for instruction 
in the order that they will ordinarily be performed on the job. Concepts 
(information) to be taught can then be associated with a pertinent task 
and taught in the order determined by the sequence of tasks. Second, 
when the work role is comprised of specific but relatively unordered, in- 
terdependent tasks, the tasks can be sequenced in accordance with the 
presumed or proven demands of prerequisite competencies; that is, the 
results of all the task analyses can somehow be combined such that all 
the contingency relationships are taken into account. Third, in even 
more complex work roles perhaps the criteria for task sequencing should 
be the development of a pre-specif ied conceptual structure. In this event, 
a model of the appropriate conceptual structure could be identified, se- 
lected tasks might then be related to the conceptual structure, and the 
tasks (as vehicles for instruction) sequenced to permit an inductive or 
deductive approach to the development of the desired conceptual structure. 
Thus, unlike the first procedure suggested in which tasks are ordered in 
accord with the principle of job performance, the third procedure would 
order tasks so as to develop a predetermined conceptual structure. Con- 
ceptual structure development would be the sequencing principle applied. 

Finally, the instructional program might not necessarily have to 
utilize only one of the suggested sequencing procedures. Cooperative 
programs in engineering, for example, typically apply at least two pro- 
cedures: on-the-job experience and directly-related instruction are se- 

quenced according to the dictates of the job, while the more "theoretical 11 
in-school instruction is sequenced according to some "other" principle. 
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VII, Miscellaneous 



Subsequent steps in the curriculum development process, such as identi- 
fying conditions of learning, designing an instructional strategy, developing 
instructional events, and creating student and curriculum evaluative pro- 
cedures and devices were not discussed during the seminar. Instead, atten- 
tion was given briefly to the mechanics of bringing about educational change. 

Large-scale, "wholistic" attempts at change seem to be called for. 
Whenever possible, the scope of curriculum innovation should be as broad 
and as coordinated as possible in order to maximize its impact upon the 
student and thereby improve the likelihood of making significant educational 
improvements. Where dollars are limited, the scope of the curriculum to 
be changed may have to be restricted, but each such attempt, regardless of 
its scope, must provide for concurrent change in related support systems, 
such as pupil-personnel, teacher education, and administrative services. 
Failure to provide for change in relevant back-up systems has often been 
the apparent reason why otherwise promising innovations have failed to 
become regular parts of the school’s program. 

Some of the curriculum changes seen as desirable by the seminar mem- 
bers include (a) increasing the attention to be paid to the "products" of 
instruction, (b) making education more task oriented, with the tasks having 
face validity for students and actual relevance to out-of-school behavior, 
and (c) permitting more non-verbal behavior in the learning process, es- 
pecially at the earlier stages of schooling, in order to provide the 
"concrete" experiences necessary for later abstract conceptual development. 



VIII. Conclusions 



The seminar was successful in highlighting some of the complexities 
in the process of developing manpower training curriculums, but it also 
served to illustrate that through cooperative research efforts the problems 
may not be insurmountable. Both military and non-military personnel are 
working towards the creation of objective, empirical procedures and tech- 
niques which will facilitate curriculum development efforts. 

While it would be presumptuous to assume that the seminar produced 
any lasting solutions to curriculum development problems, it was able to 
begin to formulate a series of rather discrete steps in the development 
process and to suggest some of the problems related to each step. For 
example, it seems quite clear that one of the major problems facing vo- 
cational educators is to specify precisely the role which students are 
expected to play after completion of training. Further, it seems equally 
clear that while significant progress has been made in identifying and 
clustering technical tasks, considerably more research is needed to (a) 
develop task sampling frames from which it will be possible to select the 
tasks to be included In the training program, (b) precisely define the 
conditions and performance requirements for each task, and (c) propose 
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new schemes for organizing the tasks in the most efficient instructional 
sequence. 

In conclusion then, the ideas presented by the discussants and sum- 
marized here may not be a revelation to the profession, but they are at 
least enlightening in that a framework is provided to systematically de- 
scribe the nature of the conceptual and operational problems facing 
curriculum developers. Through continued cooperation between military 
and non-military researchers, it may be possible to study and solve some 
of these problems. 
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A SYSTEMS APPROACH TO CURRICULUM IMPROVEMENT 



David S. Bushnell^ 
Battelle Memorial Institute 



Introduction 



Education in the United States is now a $65 billion enterprise* 
According to President Nixon , we now spend as much money on education in 
this country as does the entire rest of the world. Viewed from the per- 
spective of the return on the dollars invested in education! economists 
have recognized that such expenditures may account for a large share of 
the economic growth achieved in this country in recent years. Education's 
role in maintaining and enhancing the skill and adaptability of a nation's 
labour force and in providing the wherewithal for continued economic growth 
through research has made it possible for us to continue our rate of 
economic advancement and growth. Paradoxically, our public educational 
system which has served us so well in the past now suffers from the in- 
firmities of institutional rigidity. The rapid strides of a computer 
based technology, the rising cost of education, the growing impatience of 
the younger generation with outmoded, non-relevant curriculum, and the 
skepticism of the disadvantaged with the prescriptions of the professional 
educator provide evidence for the fact that educational institutions have 
not kept pace with these demands. This year's rash of riots and protests 
have put educators on notice that schools must begin to adapt to the needs 
of its students In ways which are appropriate to their short-term and long- 
term expectations. 

We know how to improve the effectiveness of education but we're not 
quite sure how to go about it. The problems and shortcomings indicate 
that there is a need for clearer definition of goals as well as a means of 
dealing in a coordinated way with the multitude of variables which relate 



^This paper was prepared during a one year appointment as an Advanced 
Study Fellow with the Battelle Memorial Institute. 
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directly to the achievement of those goals. Past efforts at school 
improvement have tended to deal with only fragmented pieces of the total 
learning structure. Each of the subsystems making up that structure must 
be viewed as part of a total system. Matt Miles observes that many experi- 
mental programs within the larger system fail to be exploited because 
the particular innovation is perceived as creating too many ripples, too 
much pressure on the other outmoded practices, with the result that those 
having vested interest in the conventional procedures resist the improve- 
ment . 



For this reason, small scale curriculum improvement efforts such as 
many of those currently under development in vocational education, are 
likely to have little impact because they deal only with a small part of 
the total learning process. What is needed is a more massive (both in 
funds and effort) approach where attempts at piecemeal improvements are 
linked together across a spectrum of subject matter areas and grade levels. 
Few attempts until recently have been designed to go beyond the boundaries 
of a specific subject matter area when concerned with curriculum reform. 
Even fewer adventuresome souls have teen willing to consider the need 
for modifying curriculum content, instructional procedures, teacher roles, 
and administrative procedures, all at the same time! Yet that is probably 
what is required if significant improvements in student performance and 
satisfaction are to be brought about. 

Dr. Lloyd Humphreys, Chairman of the Department of Psychology at the 
University of Illinois in a presentation before the 1969 American Psycho- 
logical Association Convention listed four priority items which he felt 
should be given our full attention in any attempt to improve the effec- 
tiveness of schools. His first principle was that curriculum improvement 
offers more payoff potential than any other form of educational change. 
Secondly, he argues that appropriate use of contingency reinforcement 
schedules will dramatically aid in the internalization of the child's 
desire to learn. Third, he stresses the importance of accurately measuring 
academic ability and adapting learning materials and requirements to that 
ability. Intelligence, he recognized, is a function of intellectual 
skills, knowledge, learning sets, and generalization tendencies. Each 
ingredient is interdependent with the others and resistant to chance in 
the short run. And fourth, Humphreys contends that the time the student 
has to learn or is willing to invest in learning may be as important a 
determinant of his ultimate proficiency as any other part of the teaching 
strategy employed. I mention these four principles only to keep in per- 
spective where we as conferees might spend our own time most productively 
during these next two days. Which issues to examine, what variables to 
explore, and where to move next, these questions deserve at least pre- 
liminary attention. We should strive for the wisdom to understand and 
the courage to accept that which we cannot change (to paraphrase an old 
adage) . 

Fortunately, there is a relatively new analytical approach which 
offers us a way of manipulating the important learning vrriables and 
linking their results to the ultimate criteria of improved student per- 
formance. Systems analysis of course is not a new or magical formula for 
guaranteeing success. Most of us are well aware of the benefits of 







1 

- : ^^> > ^*- i «‘ , ?t^ , !!i%6W«3W^ *”**■ •‘>‘* rf •••«^^*~-%^*''^>f--r^^ir^ t^^^-j*j«' ! A''u^#aswit?*?''5 ; = ^ ‘ r f 



- 13 - 



applying this technique to the design and implementation of training 
programs in military and industrial settings. Task analysis and job 
clustering have contributed (or will contribute) greatly to the prepa- 
ration of students and adults for viable careers, provided we are success- 
ful in keeping in focus the necessary interplay between the adaptive, 
functional, and specific content skills which comprise all job require- 
ments. How to translate these into vocational curriculum content is the 
central concern of my paper. 



Is A Systems Approach in Education Possible? 



Before attempting to argue that systems analysis techniques have a 
legitimate place in vocational curriculum design, let me observe that 
there are significant differences between an educational system and other 
systems. Introduction of the new math into a classroom generates a whole 
host of problems which are unlikely to be found in other institutional 
settings like the military. Students can’t grasp the new terms and parents 
are frustrated because they can’t help. Teachers are frustrated because 
they haven’t been taught how to cope with wide-ranging student needs and 
abilities. Principals are frustrated because their students fail to 
measure up on standard achievement tests which haven't been overhauled to 
reflect the new approach. Regional accrediting groups require time and 
empirical evidence in order to decide whether or not the new curriculum 
meets their sometimes outmoded criteria of excellence. 

There are still other problems. School boards in their worry over 
increasing demands on the tax dollar are often compelled to make hard 
choices between new course offerings and getting by with existing ones. 

This is particularly true of vocational education because the establish- 
ment of a new vocational training course can be much more expensive than 
the cost of a new physics or math program. The advantages of a new cur- 
riculum may be hard to determine empirically. Objective evidence supporting 
the choice of one set of materials compared to another may be non-existent 
(given the publisher’s reluctance to adopt or reveal performance criteria). 

The military training command, on the other hand, is training for a 
specific level of competence in a specific job which is not likely to 
change in the near future. Until recently at least, dollar and other 
necessary resources have not been a primary concern. Most trainees are 
high school graduates, well motivated, and already certain of their occu- 
pational pursuit during their period of service. Shifts in task require- 
ments are usually anticipated well in advance of demand. Proficiency of 
the job incumbent following training is easily monitored with feedback 
to the training command a certainty. 

This brief discussion is meant to establish how the level of complexity 
differs between the centrally-controlled and run military or industrial 
establishment and the decentralized and locally-managed school establish- 
ment. To realize this difference may be to realize the problems peculiar 
to applying a systems analytic approach to curriculum improvement in public 
education. 
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It is difficult to imagine anything or anyone not functioning in 
some kind of system. You have only to think of the human body as a series 
of interrelated systems and subsystems to gain a feeling for the compre- 
hensiveness of this approach. The person or planner who applies a systems 
approach will usually view the institution or organism as a functioning 
entity. The basic elements are to set forth one's goals in quantifiable 
terms, plan and present the various alternatives to achieving these go^ls, 
operationalize the plan to be carried out, evaluate the results and feed- 
be vk the evaluation into the system so that the operations can be 
appropriately modified or revised. Morgan and Morgan, in their article 
on "Systems Analysis for Educational Change, state that "the first step 
in applying a systems approach to education involves the specific defini- 
tion of what outcomes or results are desired. It is against these specifi- 
cations that the system, whether spaceship or educational program, is to 
be built. The next step is to analyze the many variables that will con- 
tribute to the performance of the system. What are the means and strategies 
that can be employed to produce tne desired outcomes? This analysis of 
means can be complicated or relatively simple. It involves the study of 
relative cost of means, feasibility of their use, and the predictability 
of the results." 

Modifying an ongoing educational program has oftentimes been stymied 
by our inability to cope with the multiplicity of variables involved. It 
is analogous to trying to convert a locomotive from coal to diesel fuel 
without stopping the train. Few educators, until recently, were capable 
of even verifying which educational procedures were the most valid. The 
availability of computers to education and the more sophisticated applica- 
tion of systems analysis have recently made it possible to deal in a 
systematic way with these highly complex problems that require simultaneous 
juggling of multiple variables. As an example, let's focus on the prob- 
lems of identifying performance objectives in vocational education as a 
means of building and integrating curriculum materials so that they might 
more easily be adapted to individual student requirements. 

One of the first and most important tasks to be undertaken in sys- 
tematically building an individually oriented vocational curriculum with 
the necessary variety of materials and pathways needed to accommodate stu- 
dents of varying abilities, interests, and learning styles would be to 
convert where possible specific work tasks into observable behavioral 
specif ications, i. e. , a catalog of appropriately classified performance 
objectives that would describe with precision the minimal levels of per- 
formance that each trainee should be able to achieve. This does not ignore 
the fact that some trainees could go well beyond the minimally sufficient 
level. However, it would describe the adaptive, functional, and specific 
content skills needed to qualify for a given occupation. 



2 

Morgan, Robert and Morgan, Jack, "Systems Analysis for Educational 
Change," TREND, Spring 1968, pp. 28-32. 




Why Performance Objectives? 



Apart from a growing discontent with the traditional ways in which 
performance objectives have been stated, what compelling reasons are there 
for going to the very great trouble of developing a full inventory of 
appropriate skills? 

First, in using the systems approach to curriculum design, the goals 
or objectives for the program must be stated in terms of output specifi- 
cations . In education, these specifications can oftentimes be stated in 
terms of behaviors. Without them there is little basis for deciding 
which learning intervention or teaching strategy would be most effective. 
When decisions on the selection of teaching strategies have been made with- 
out performance objectives, there are no empirical means of determining 
the degree of their effectiveness. It is doubtless true that most de- 
cisions about changes in instructional practice are made without verifica- 
tion in terms of what someone thinks is the effect of the practice on what 
is hoped to be the result. Consequently, systematic program revisions 
are virtually precluded in the absence of measurable performance objectives. 

A second reason relates to the need for longitudinal validation of 
the effectiveness of public education in preparing young people to cope 
with their social and economic environment after they leave school. Un- 
less we know with what behavioral attainments a youngster enters the adult 
world, there is little basis for relating his success or lack of it back 
to his school experience. The selection of given sets of performance ob- 
jectives is purely judgmental. People of wisdom must decide that one ob- 
jective is important and reasonable and that other objectives are not. 

At least one important criterion for the selection of objectives is that 
they appear to relate to preparing a student for his adult roles — as a 
worker, parent, or as a citizen. Such objectives tend to be selected in 
terms of their face value. That is, they look as if they ought to relate. 
Face validity is often misleading, and the appearance of relevance may 
not be supported by correlational analysis. Revisions in educational 
goals as a result of follow-up studies of high school students ought to 
be, but are rarely, done. 

A third reason for requiring performance objectives is the need to 
assess the cost-effectiveness of educational programs, The American tax- 
payer will inevitably grow weary of continuing to vote for increased tax- 
ation for education with no tangible evidence of the effect that these 
expenditures have on the education of his children. With more precisely 
stated educational goals, it should be possible to associate behavior 
change with program costs. Student learning should certainly be the main, 
if not the only, basis upon which cost-effectiveness analyses should be 
made in education. 

The most important reason, however, is the need for a detailed, 
carefully sequenced set of performance objectives and related achievement 
measures to help insure the adaptability of the curriculum to individual 
requirements. If the schools are ever to provide learning experiences 
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tallored to an Individual student's measured progress and tied to what a 
student last learned, his step by step progress In mastering an Instruc- 
tional sequence must be measurable. 



Problems of Measurement 



If It follows that developing performance objectives Is a worthwhile 
and necessary step in the design of a sequenced series of learned units, 
then the problems of doing the job should be considered. Preparing educa- 
tional goals in behavioral terms is not altogether new, but scholarly 
dialogue on the subject is remarkably thin. One of the most definitive 
works on the subject is probably Robert Mager's Preparing Instructional 
Objectives . * Without taking anything away from Mager's substantial con- 
tribution, the paucity of discourse on the problem of stating performance 
objectives in all spheres of learning has failed to provide ready answers. 
People who have written behavioral objectives (and there have been a fair 
number) have generally followed the procedures set forth by Mager. Yet, 
it appears that a large number of the most important educational goals do 
not lend themselves to objectification in the Mager manner. Consider 
citizenship, social skills, and aesthetics, for example. 

In a recent educational meeting, David McClelland raised the objec- 
tion that much of the work done on educational objectives was limited to 
"respondent" behaviors. He defined respondent behaviors as those elicited 
in a contrived stimulus situation which were appropriate for some parts 
of education, but not for all. He described the need for means of objec- 
tifying and measuring "operant" behaviors. Operants are defined as re- 
sponses emitted by the learner in the absence of any clearly defined stimu- 
lus situation. For example, if an educational goal is to have the student 
learn to appreciate and enjoy good literature, the best test of whether 
or not this aim had been accomplished would best be assessed by the way 
in which the student actually invests his leisure time outside of school 
hours. The present technology of behavioral objectification and measure- 
ment doesn't really accommodate these kinds of "real world" criteria. 

Another significant problem in behavioral objectification is the 
horizontal and vertical articulation needed among the conventional subject 
matter areas. A number of past efforts have not met with great success. 



^Mager, Robert F. , Preparing Instructional Objectives . Palo Alto, Calif. 
Fearon Publishers, Inc., 1962, 60 pp. 

^See, for example, Stanford Research Institute's careful evaluation of 
the well publicized Richmond project, (BR5-1337) Kincaid, Harry, A Pre - 
liminary Evaluation of Richmond Plan (Pre-Technical Plan ) Programs in 
Secondary Education , S.R.I., Menlo Park, California, 1968. ERIC Docu- 
mentation No. 025654. 
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One of the principle reasons for this lack of success is that the integra- 
tion has been attempted at the level of the teaching process without 
sufficient attention to the integration at the level of the educational 
objectives. An effective curriculum requires an appropriate fusion or 
integration of the objectives across subject matter lines — a task for 
which there is little precedence and one which calls for the careful phasing 
of a series of developmental activities. 

In seeking to develop integrated performance objectives, the groups 
and individuals qualified to do this work inevitably move to a subject 
matter orientation as they plan their projects. The way the learning pro- 
cess has been organized, by discipline, with relatively little exchange 
across disciplines, may not be the most efficient way of arranging the 
learning experiences. It is predicted that a careful analysis and evalu- 
ation of the performance objectives of our present day secondary school 
curriculum would lead to a substantial reorganization of the objectives. 

It is possible, even likely, that there are wasteful redundancies in 
teaching the same or similar objectives in several fields. 

More important than redundancies, however, are the gaps. That is, 
there are important educational objectives which are assumed to be taught 
somewhere in the curriculum but, in fact, may not be taught anywhere. 

There may be essential educational goals and objectives which have fallen 
through the cracks one finds between disciplines. In the interest of 
efficient learning, it may be more sensible to reclassify certain of the 
objectives into new groupings that are independent of the disciplines from 
which they were originally derived. For example, the basic principles of 
science may be better taught to some students in an industrial arts or a 
home economics setting than in the science class. 



Classification of Performance Objectives 



Considerable attention has been given by educators to classification 
strategies and educational objectives. ^ These taxonomies have sometimes 
been developed independent of a careful and complete analysis of the ob- 
jectives. A more logical and scientific approach to building a taxonomy 
would be to first define all or as many as possible of the performance 
objectives in our present-day curriculum, then carefully evaluate and 
classify these objectives. The taxonomy would then be derived from this 
analysis, grouping objectives in terms of learning process or problem- 
solving skills. 

Not all learning "outcomes 11 can be defined in terms of specific ob- 
servable behaviors. A large part, however, can and should be objectified. 



O 



^Much of the remainder of this paper draws heavily upon the writings 
and thinking of Dr. Bruce Tuckman, Rutgers University, who is presently 
directing Project SCOPE. 
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The following steps are suggested as a procedure for systematically re- 
ordering the traditional subject matter Into a more "organic" curriculum: 

Step I - Development of Objectives by Sub j ect Matter Groupings 

It is essential that discipline scholars, classroom teachers, cur- 
riculum supervisors, and behavioral technologists be trained to work to- 
gether in this initial phase of specifying performance objectives. 

Writing performance objectives is a demanding technology and a skill in 
short supply. A team effort requires a number of specialists who, never- 
theless, have a common understanding of the procedures involved and 
commitment to the operationalizing of instructional objectives by subject 
matter groupings. 

Step II - Analysis and Integration of Objectives 

The second phase requires the analysis and integration of objectives 
independent of the originating disciplines. Two centers^ have recently 
been set up with USOE funds to inventory, classify, synthesize, and re- 
furbish performance objectives for operating schools. It is hoped that 
both will independently achieve an empirically sound classification scheme 
to be employed in the subsequent redesign of curriculum materials. 

The types of services to be performed in the coordination and syn- 
thesis of performance objectives should be as follows: 

1. Receive sets of performance objectives in various subject matter 
areas and alter the language employed, when necessary, to guaran- 
tee that each is truly "behavioral." In accomplishing this, a 
glossary of behavioral words such as the following could be used: 



0 . 


identifying 


f. 


describing 


b. 


distinguishing 


8- 


stating a rule 


c. 


constructing 


h. 


applying a rule 


d. 


naming 


i. 


demonstrating 


e. 


ordering 


J. 


interpreting 



'■The first Center is under the direction of Dr. Bruce Tuckman at Rutgers 
University. The second Center is located at UCLA under the direction 
of Dr. James Fopham. For a more complete discussion of the role of 
the second Center, see Popham, W. James, "Focus on Outcomes — A Guiding 
Theme of ES *70 Schools," Phi Delta Kappan , Dec., 1969. 
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By using a constant vocabulary to describe the "behavior" 
called for in an objective, one achieves not only greater stan- 
dardization, but one builds in an automatic basis for coding and 
classifying. 

Thus, the first task will be to "clean up" the objectives 
and apply a constant "behavior language." This will entail not 
only the reproduction of a multitude of objectives on cards and 
storing them, but also coding . The first task will also include 
the refinement and complete description of the action word glossary. 

2. Code all behavioral objectives. Two reasons support this pro- 
cedure: (a) a cataloging device is needed which will provide 

for easy retrieval; (b) a way of reexamining and reorganizing 
the objectives along different lines of commonality is needed 
to serve as a basis for curriculum building or revision. 

Two coding systems need to be built into the objectives. 

The first is the level of generality of the objective as seen by 
the writer of the objective (e.g. , 1.11 Science, Physics, Ohm’s 
Law). A subject-matter classification scheme — using three or 
four levels of description — neer’j -,o be developed and used by 
the subject-matter specialists. Second, a glossary of behavioral 
terms, described above, need to be developed and refined and used 
to classify each objective. 

3. Classify each objective in terms of the process or activity de- 
scribed and the object of the process. This scheme has some 
similarity to the Bloom Taxonomy, ? but will differ in being two- 
dimensional rather than unidimensional. The classification of 
performance objectives by subject matter areas while only an inter- 
mediate step should have immediate value for teachers and curric- 
ulum planners throughout the country. An example of such a 
classification scheme might appear as follows: 



OBJECT 

12 3 4 5 

PROCESS Symbols Ideas Things People Self 



1. Perception 












2. Conceptualization 












3. Application 












4. Evaluation 












5. Manipulation 













^Bloom, Benjamin S., Taxonomy of Education Objectives , New York; David 
McKay Co., 1956. 



A MANPOWER DELIVERY SYSTEM: 
IMPLICATIONS FOR CURRICULUM DEVELOPMENT 



Robert G. Smith, Jr. 

Human Resources Research Organization (HumRRO) 



Introduction 



Once you have become infected, as I have, with system thinking, you 
develop some special habits. You begin thinking about purposes, about 
functions to carry out those purposes, and about interrelationships among 
the functions. But especially you tend to believe that the overall con- 
text of a matter is important. 

Following this belief, I plan to present a simplified and abstract 
model of a manpower delivery system. I hope that this model will enlighten 
our later discussion by providing a suitable context for the translation 
from the military problem to the civilian problem. 

The features of this model have been suggested by a study of the re- 
port of planning for the State of Pennsylvania by Arnold (1969). 



Occupational Demands 



In Figure 1 is shown one of tha major components of the manpower 
delivery system. These are a set of occupations. The reader should think 
of the block of occupations as representing a time line and changes in 
occupations in accordance with various trends. For instance, at the top 
we see an occupation represented which has a relatively small requirement, 
but a requirement which is expected to be stable over the next several 
years. Moving dovr. the block we see an occupation that has a larger re- 
quirement right now, but one which is declining in demand. Next, we see 
an occupation which has a fairly small demand at the present time, but 
whose demand is expanding. At the bottom of the occupation block we see 
a representation of an occupation with a large and stable demand. 
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From this Figure it is clear that when we consider the implications 
of both stable and changing demands, we must develop information con- 
cerning the overall economic predictions for the industries in which these 
jobs occur, and ve must also be very much concerned with changing demands 
for various kinds of occupations. 



Educational and/or Training Institutions 



Figure 2 adds the next major group of components to the system. 

Here we have a set of educational or training institutions whose purpose 
it is to provide manpower to the various occupations. Again, it seems 
reasonably obvious that the educational and training facilities should 
have a quantitative relationship v>ith the demand for occupations. Thus, 
if the occupations are decreasing, the education or training institutions 
should decrease their output. If the occupational demand is increasing, 
then the education or training institutions should increase their flow. 

Of course, it is probably easier to start new courses than it is to stop 
or reduce existing ones. 

In addition to these quantitative considerations, analyses should 
be made of the occupational demands it; a qualitative sense to determine 
what kinds of performances they will demand of the people who enter them. 
This is what will determine the objectives. 



Guidance 



In Figure 3 we add some additional features to our model. The first 
of these is guidance. By guidance I refer to the activities of attempting 
to provide some reasonable match between the characteristics of the 
individual student, the demands of various occupations, and the educational 
barriers which must be crossed before the individual can enter into his 
preferred occupation. 

If the term guidance is too directive for some readers, they might 
wish to substitute the term counselling for it. 

The guidance activities represent a relatively long-range method by 
which quantitative adjustments could be made to adjust the flow of 
personnel entering various occupations to the long-range trends for that 
occupation. At the same time, it would attempt to insure a reasonable 
degree of success for the individual during the training or education for 
the occupation, as well as success and satisfaction in the occupation 
itself. 

Among the activities which I would lump under the function of 
guidance would include: the measurement of student aptitude and interest; 

both general Introductions to the world of work, and specific introductions 




to the particular occupations the student Is interested in, and the 
charting of an educational path which will help the student to enter his 
preferred vocation or profession. 



Placement 



If guidance represents the long-term adjustment of the system to the 
changing demands of various occupations, placement represents a relatively 
short-term adjustment. It refers to those methods by which a match is 
obtained between the vocational or professional graduate and the occupation. 
Placement would include such activities as those of the state or federal 
employment services, the activities of educational institutions themselves, 
and the activities of the individual himself. 

A number of studies have shown that most people find their own jobs, 
perhaps with help from friends or family, rather than using governmental 
or educational placement activities. This suggests that instruction in 
job seeking could be very useful. 

I remember when I was a budding new Ph.D. no one told me about the 
placement activities by which psychologists obtained jobs, such as the 
hiring halls at the professional meetings. I obtained a job through one 
of my professors. It was an excellent position, and a very interesting 
one. I don't regret having taken the job, but I have* sometimes wondered 
whether I would have taken it if I had had a broader range of choices 
from which to look. 



Barriers 



There are a great many occupational barriers which lie between the 
educational or training institutions and entry into the occupation. 

Examples of barriers would be various kinds of licensing requirements 
(often with little relevancy to job success) and union membership, especially 
if the unio.. has a policy of restricting membership. 

Some of these batrlers may be very relevant ways of assuring that 
only qualified people enter the occupation. Other barriers may be highly 
irrelevant. For instance, city government jobs may have the requirement 
that the person have lived in the city for a given length of time. This 
would restrict the possibilities fo* employment of someone who was a 
resident of another place. 

It seems reasonably obvious that whatever barriers exist should be 
identified, and the appropriate action taken to be sure that the student 
will not encounter a barrier in his progress through the system. 

The existence of barriers also has implications for the evaluation 
of vocational education programs. These programs frequently are evaluated 
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in terms of the number of students placed. If we have a veiy effective 
educational institution from the standpoint of imparting the necessary 
job skills but yet the student cannot get past a barrier, this is not a 
suitable way of evaluating the excellence of the instruction. At the 
same time, it does suggest that the system needs to be looked at from the 
standpoint of making sure that the barrier is actually necessary, or that 
only those students who could get by the barrier are being guided into the 
appropriate educational programs. 



Information Flow 



Figure 4 now completes the model by adding arrows which represent 
the needs for information flow between the various parts. For instance, 
information from the various occupations is needed by the planners of 
educational and training programs so that the size of various programs 
can be adjusted to changing demands of the world of work, and so that 
qualitative trends in occupations can be reflected in the educational 
objectives. Information from the occupations is also needed by the 
guidance activities. The information needed by the guidance activities 
is clearly of a reasonably long-term nature. It should include some 
consideration of the aptitude requirements of the occupation or its 
educational counterparts, as well as the early special characteristics 
of the occupation that would be of interest to students and the long-range 
quantitative trends in the occupations. 

There is also a requirement for the placement activities to obtain 
information concerning Job vacancies and related information of a short- 
range nature, in order to carry out the placement activities in an efficient 
manner. 

There clearly is a requirement for the guidance activities to provide 
feedback to the education and training institutions on the characteristics 
of prospective students. 

This model has been an abstract and rather generalized model but it 
does serve the function of pointing out to educational and training planners 
the principal things to look at when conducting a system analysis for vo- 
cational or professional education. Obviously, the specifics may vary 
from one state to another, or from one employment area to another. 

Yet, the general points should be similar. 

REFERENCE 
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IMPLICATIONS OF AIR FORCE OCCUPATIONAL RESEARCH 
FOR CURRICULUM DESIGN 



Raymond E. Christal, Chief 
Occupational and Career Development Branch 
Personnel Research Division (AFHRL) 
Lackland Air Force Base, Texas 



Introduction 



For the past twelve years, the Occupational and Career Development 
Branch, of the Personnel Research Division (AFHRL), at Lackland Air Force 
Base, Texas, has been developing methods in such areas as job analysis, 
job evaluation, work organization, career planning, reassignment systems, 
performance evaluation, and job requirements. We are concerned with the 
study of individuals after they leave technical training schools, and we 
give particular attention to the occupational framework within which 
these individuals function. Our research has already had a significant 
impact on technical school curricula, and we are currently undertaking 
the development of methods for translating occupational information into 
a form specifically designed to help individuals who are responsible for 
curriculum definition. I will discuss this work in some detail. I will 
also describe two mathematical models which we have found to be very use- 
ful in our research, with the hope that they may be of some value to you. 

Factors Affecting Curriculum Definition in the Military Services 

Before getting into the main discussion, let me pinpoint some factors 
that make curriculum definition in the military services a little differ- 
ent than it is in the civilian sector. 

First , a majority of individuals entering the services have already 
graduated from secondary school. 

Second , most of the technical training courses given by the military 
services are of short duration — usually covering a period of between 
eight and fifty- two weeks. 

Third , when a man in the service is sent to a technical training 
school, the decision has already been made that he is being trained to 
perform work in a particular occupation. Except under unusual circum- 
stances, this decision is not subject to change. I am led to believe 
that occupational choice for those entering vocational and technical 
training in the civilian sector is not nearly so stable or specific. 
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Fourth, in the military services we do not have to protect individ- 
uals against their skills becoming obsolete. If a type of job disappears 
from the inventory, the military service foots the retraining bill, and 
the individuals involved suffer no loss of pay. 

Fifth , the services often have years of advanced warning before jobs 
are established requiring new occupational skills. For example, curricu- 
lum experts begin planning courses to support a new weapon system while 
the new system is still on the drawing board. 

Finally , technical training for service personnel does not end when 
they leave the school setting. They receive formalized on-the-job train- 
ing, and they return to school for advanced training when such training 
seems warranted. 

I would judge that all of these factors tend to make curriculum 
definition a little easier for those in the services than it is for those 
who work with vocational and technical high schools. But I would not want 
to leave you with the impression that the military services have the curri- 
culum problem licked. Far from it. Data collected during the last several 
years make it apparent that problems associated with "what to teach" are 
as critical as problems associated with "how to teach," even in the mili- 
tary services. 



Military Needs for Job Information 

Let me describe the situation a little more precisely. I'll use the 
Air Force for purposes of illustration. 

In the Air Force, there are about 800,000 enlisted personnel perform- 
ing thousands of different types of Jobs at hundreds of locations through- 
out the world. Each of these Jobs has been established by seme local 
cotmander who can, within limits, assign work to men any way he sees fit 
in order to accomplish his mission. It is the function of the Air Force 
training establishment to provide individuals with the knowledges and 
skills they will need for performing the Jobs they will encounter. This 
means that the local commander needs some way to communicate the nature of 
the jobs he has established to trainers. Until recently, his only means 
for communication was through the assignment of a five-digit specialty 
number to each job, classifying it into one of approximately 250 career 
ladders, at the apprentice, Journeyman, first-line supervisor, or superin- 
tendent level. This system is grossly Inadequate, because it does not 
provide information at a sufficient level of detail for curriculum design- 
ers. In the first place, there are many more types of jobs than there are 
specialty numbers. New jobs are constantly being created in the field, 
but they nevertheless must be classified into the existing classification 
structure. Individuals who are responsible for the classification struc- 
ture would like to maintain an appropriate classification code for each 
job, but they have no move Information than the trainers about the nature 
of jobs being assigned to personnel. Individuals maintaining the classi- 
fication structures and those maintaining course curricula each have to 
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depend upon complaints from the field for indications that the system needs 
modification. The training command has course evaluation teams which visit 
a few organizations to talk with supervisors about the problems being en- 
countered by course graduates; but this type of inquiry is both expensive 
and extremely limited in terms of the kind and amount of information 
yielded* 

Because of this serious lack of information, the first goal we estab- 
lished for our occupational research program was to develop techniques by 
which the Air Force could collect more detailed and specific information 
about the jobs being assigned to personnel. We finally settled on a tech- 
nique involving the administration of job inventories to workers in the 
field. 



Description of Job Inventory 



Now let me describe a job inventory to you. It contains two sections. 
The first section consists of background questions to be answered by a 
worker about his job and himself - - - questions relating to previous edu- 
cation, time-on-the-job, tools utilized, equipment worked on, interest in 
job, pay grade and so on. For an inventory being administered to civilian 
employees, one might wish to collect information about union membership, 
working hours, size of organization, products manufactured, and job loca- 
tion. Any questions can be included in the background information section 
of an inventory which will help answer questions about the occupational 
area . 



The second section of a job inventory is simply a list of all of the 
significant tasks performed by workers in the vocational area being sur- 
veyed. That is, it includes tasks being performed by apprentices, jour- 
neymen, first-line supervisors, and superintendents in a single occupa- 
tional area, such as automotive repair, firefighting, or metal working. 

If this task list is properly constructed (and this point is important to 
understand), then every worker in the occupational area should be able to 
adequately define his job in terms of a subset of tasks in the inventory. 

In the Air Force ve normally administer an inventory to several thou- 
sand workers in the heavily populated career ladders, and we attempt to 
obtain 100X samples in the less populated areas. A worker fills out the 
background questions, checks the tasks he performs, writes in any signifi- 
cant tasks he does that are not on the list, and indicates how his work- 
time is distributed across the tasks in his job. These data are sent to 
a central agency for computer analysis. 

Advantages of the Job Survey Approach 

Before I describe how we make use of this information, let me mention 
some of the reasons why we like the job inventory survey approach. First , 
the technique is economical. Data can be collected from thousands of 
cases throughout the Air Force for less than it would cost to collect data 
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from a few cases using professional job analysts. Second , the informa- 
tion is quantifiable. That is you can actually count the number of 
people performing any given task, and describe their characteristics. 

Note that data collected by job analysts are not quantifiable. No two 
analysts will describe a job in exactly the same terms. The fact that 
information collected with task inventories is quantifiable means that 
it can be stored, manipulated, analyzed, and reported by computer. The 
fact that it is quantifiable also means that it can be subjected to re- 
search — that is, it can be validated and checked for stability using 
conventional statistical techniques. 

This is all well and good, but the $64 question is whether individual 
workers are honest and conscientious when they fill out job inventories. 

Is the information they provide of good quality? 

We feel satisfied that the answer to this question is "yes" — at 
least as far as Air Force enlisted personnel are concerned. We know that 
when a worker fills out an inventoiy on two occasions, he gives essen- 
tially the same information both times. Split-half correlations for in- 
formation such as the percentage of employees performing each task run in 
the neighborhood of .95 to .99. Supervisors agree with the information 
provided by their subordinates. Information collected with dally work 
records is consistent with information collected with inventories. Work- 
ers do not inflate their job descriptions in terms of the number or diffi- 
culty levels of tasks reported. There is a high probability that signi- 
ficant tasks missing from the inventory will be written in by workers who 
perform them. In short, we feel that the technique yields information of 
good quality. 



Analysis of Job Survey Data 



Once we have collected occupational information from a large sample 
of individuals, the resulting data are fed into a computer. A record is 
generated for each individual case which contains his identification in- 
formation, his answers to the background questions, and the amount of 
time he spends on every task in the inventory. Thus, we generate a bank 
of information describing the Individuals and Jobs in one occupational 
area. 



In the time we have 1 cannot begin to describe the ways in which we 
analyze this occupational information to serve the needs of Air Force 
managers. We have developed a comprehensive set of computer programs, 
called CODAP, which is in essence an occupation information retrieval 
system. It contains dozens of programs, each designed to analyze occupa- 
tional data to provide a manager with the information he needs for making 
a particular type of decision. What 1 propose to do Is to describe a few 
of the outputs which are normally obtained by individuals responsible for 
training courses in the Air Force, and to indicate how these outputs have 
Influenced curriculum design. 




Consolidated Job Descriptions 



The first program, which Is simple but extremely useful, enables the 
investigator to produce a consolidated description of the jobs! being per- 
formed by any group of workers of interest. For example, one might wish 
to see a description of the jobs performed by individuals in a given geo- 
graphical area who repair a particular type of equipment, and who have been 
on the job less than one year. A consolidated job description can be ob- 
tained for any group of workers which can be defined in terms of informa- 
tion in the background section of an inventory. 

Appendices I and II present typical duty and task descriptions, 
which I often use as examples, since they cover an occupation existing in 
all services as well as in the civilian sector. They describe jobs of 
journeyman medical laboratory technicians, working in hospitals and clinics 
throughout the Air Force. 

Let's spend a little time studying the data in Appendix II. Every 
task performed by journeymen laboratory technicians is listed. The first 
task reads "collect blood specimens directly from patients." In the first 
of the four columns of numbers you will observe that 93.4 percent of these 
workers perform this task as part of their normal job. The second column 
indicates that the individuals who perform this task spend about 1.7 per- 
cent of their worktime on it. In the third column, we find that about 1.58 
percent of the total worktime available to journeymen technicians is spent 
on this activity. If you were to sum the values in this third column 
across all tasks on this job description, the total would be 100 percent, 
since every task performed by any journeyman is listed. Cumulative times 
are shown in the fourth column, so you can see that the tasks on this first 
page account for about 38.56 percent of the total worktime available to 
the group. Finally, you will notice that the tasks are listed in order of 
the time spent values in the third column. 

Use of Consolidated Job Desct iptions in Curriculum Design 

Now let's look at this job description with a view toward curriculum 
design. 

The first column of values is of special concern. It reports the 
probability that a journeyman in this occupation will be required to per- 
form each of the tasks listed. You will observe that the tasks appearing 
on this first page are encountered by 70 to 90 percent of Journeyman medi- 
cal laboratory technicians. In the Air Force, an entry level course would 
include training on these tasks. Now look at the last page of the job 
description. Here you will observe tasks which are encountered by very 
few Journeymen. Consider for a moment task M 35, which is concerned with 
conducting vitamin assays. Only about one percent of the journeymen are 
asked to perform this task, and even these cases spend only about one-half 
of one percent of their vcrktime on it. The question is whether one can 
afford to train everyone going into this field to perform a task which is 



